Security Investigation in Encrypted Environment

Nowadays, a wide variety of applications are using encryption to protect their confidential data in network communication. Since currently available security tools are mainly developed for plaintext network transmission, encryption obstructs security analysis and protection. In this thesis, we elaborate on the challenges resulting from encrypted network communication. In fact, encryption does not stop intruders from exploiting application vulnerabilities. However several protection mechanisms, e.g. kernel-based, compilerbased or third-party libraries, help to mitigate the success of attacks. We examine the feasibility of attacks against vulnerable applications on modern systems to identify the need of additional protection mechanisms. One approach is the identification of the vulnerability in order to fix it. Security testing can be applied to analyse applications with common attack payloads. For applications using encryption, the analyst has to discover the encryption algorithm and the appropriate key first, in order to proceed with testing. Another approach is to inspect the network traffic for known attack signatures. However, inspecting network data of applications with an active end-to-end encryption is not feasible on the network level without increasing the attack surface. Host-based solutions can help here, but they are limited in their use cases and prone to local attacks. Analysing binary applications without having access to the source code is known as reverse engineering. The identification of specific locations inside the application is a very labour-intensive process, but necessary for further analysis. The contributions of our work are as follows. First, we propose a new method for exploiting vulnerabilities over the network to show that system-based or application-based protection mechanisms are not sufficient. Second, we provide a framework for analysing binaries using encrypted network communication, sustaining the end-to-end encryption. The developed modules of the framework allow us to intercept, extract, modify and inject plaintext data, which is transmitted encrypted over the network. We propose a generic method to analyse applications with encrypted network communication without breaking the end-to-end encryption. Using this framework, we create a data bridge for